Current maintenance assistance technology between on-site personnel and technicians from remote locations or systems is generally difficult and time consuming in visually understanding the situation on site to determine the action needed to maintain a site systems' performance. Such maintenance assistance technology may be used, for example, in discussing vehicle issues with a road side assistance personnel or troubleshooting critical system at a power plant or oil platform in remote location.
On-site personnel are often incapable of performing maintenance based on complicated voice instruction from a remote technical expert. Also, known existing remote maintenance systems are not integrated into visualization technology to facilitate remote technical experts' ability to perform visual inspection, evaluate system performance parameter, and provide visual designator type direction back to on site personnel for troubleshooting and providing technical solutions. Existing remote maintenance systems are also not known to evaluate or address authorized versus unauthorized remote access intent to prevent outside users from intruding and/or disrupting performance of systems under test due to a lack of cybersecurity such as effective user authentication and limitations on encryption of data.
In one illustrative embodiment of the present disclosure, an operator with a wearable (e.g., a head mounted) visual user interface device and/or a device with a camera system can be positioned with respect to a system under test (SUT) with various embodiments of the invention to provide visual knowledge of the (SUT) back to a remote technical expert. The wearable or head mounted visual user interface device can also have an audio system which permits voice interaction with the wearable or head mounted visual user interface device for voice commands as well as interaction with remote experts or personnel.
Another illustrative embodiment includes a maintenance or telemetry section that collects system parameters such as environment (i.e. temperature, humidity, shock events/vibration/damage events from sensors (e.g. thermistor, humidity sensor, accelerometer, etc.) and system performance (i.e. speed, power, data transfer rates, etc.) and provides a data on such parameters and analysis of system health data based on such parameters as well as prognostics (e.g., remaining useful life estimation data analysis programming instructions tied to maintenance or system performance conditions to predict trend pattern on the condition and parameters), heuristic or expert system (e.g., if/then statement data analysis programming instructions tied to maintenance or system performance conditions to include ranges associated with such conditions and parameters) back to the remote technical expert. Image recognition and augmented reality technology can also be used to identify the system under test (SUT) and generate three-dimensional model of the system and associated subsystems with system parameters in combination with a visual user interface system.
Yet another illustrative embodiment allows a remote technical expert to access the maintenance system on site and insert one or more designators on the visual user interface presented to on-site maintenance, operator, or engineering personnel during troubleshooting or expert support activities such as flashing overlays, pointers, or other indicators.
Another illustrative embodiment incorporates a network topology to prevent remote users from accessing a system under test (SUT) through the maintenance system using, for example, separation of maintenance monitoring or telemetry systems from equipment operating or execution systems so that attacks on the maintenance or telemetry system do not impact functioning of equipment or non-maintenance systems. Embodiments of remote operation of on-site maintenance monitoring systems can be remotely operated with network design that prevents remote operation of non-maintenance systems. Authentication system and encryption methods and systems can also be included to provide cybersecurity functionality to use invention in an operational environment where data intrusion is expected.
Additional illustrative elements can include selective updating of remote maintenance by on-site systems to cut down on data traffic that can include transmissions during periods of low communications demand (e.g., 2 am local time when users are not placing demands on systems under test or communications networks used by such users or systems supporting such users), selective update of maintenance or telemetry data replication or mirroring databases at remote sites, maintenance activity based remote updates, and remote expert data requests that trigger updates or selective data transfers. Also, illustrative embodiments can also include data compression and video processing capability to reduce amount of bandwidth required to transmit both system performance parameters and video captured by the head mounted device.
An illustrative embodiment interactive remote maintenance assist system includes a head mounted device (HMD) having a visual interface, a voice interface system with voice recognition systems, a control section, an imager system operable to collect video or still images, and an input/output system. A remote maintenance server (RMS) is operable to communicate with the HMD. A control section includes a first section configured to identify the system under test (SUT) using an image recognition function, a second section configured to identify a plurality of subsystems (PLoS) within the SUT in a data library, a third section configured to create three dimensional models of the PLoS and displaying same on the visual interface of the HMD using an augmented reality function, a fourth section configured for connecting to the RMS using an encrypted connection wherein connecting is via streaming video or images sent to the RMS and connecting comprises collecting SUT data and external, to the SUT, sensor data, a fifth section configured for conducting a prognostics and/or health, maintenance, and/or management (PHM) service on the collected data to determine system health and projected health of the SUT and/or PLoS, and a sixth section configured to authenticate remote user access to the RMS, update the data library, and insert a plurality of PHM designators on the visual interface.
According to another illustrative embodiment of the present disclosure, a plurality of processing systems for an interactive remote maintenance assist system includes a first section having a first head-mounted device with a camera, a microphone, a transceiver and a power unit. A second section includes a computer processing system user input interface, a third section includes a user interface including a graphical user interface, and a fourth section includes a first storage section operable to store a plurality of data. A fifth section includes a processing system having a processing unit and a non-transitory machine instruction storage section operable to store a plurality of machine readable computer processing system instructions operable to control the processing unit and the first, second and third sections. A sixth section includes a first server system, wherein the first section is in position in a first orientation with respect to a system under test, the first section is positioned based on a first distance from a section from the system under test (SUT) based on a first field of vision determined based on a distance seen from a first camera in the first section to the system under test. The fourth section and the sixth section store a plurality of data comprising a first data identifying an authenticated user, a second data identifying configuration of a plurality of systems under test, a third data identifying configuration of a plurality of subsystems within the plurality of systems under test, a fourth data identifying a plurality of sensor parameters used to monitor the plurality of subsystems, a fifth data showing a plurality of three-dimensional models representing the plurality of subsystems, a sixth data showing a plurality of maintenance manuals reflecting the maintenance available for the plurality of subsystems, a seventh data showing a plurality of maintenance videos reflecting the maintenance available for the plurality of subsystems, an eighth data identifying a plurality of available operational modes of the system under test, a ninth data identifying a plurality of environment sensor types used to monitor the system under test, a tenth data identifying system health statuses and estimated health measures comprising remaining useful life, mean time to repair, mean time between maintenance for the plurality of subsystems and the system under test, an eleventh data identifying a plurality of videos captured by the camera in the first section, and a twelfth data identifying a plurality of images captured by the camera in the first section.
The plurality of machine readable computer processing system instructions in the fifth section may include a first plurality of machine readable computer processing instructions operable to identify a system under test (SUT) using the first section to capture a first image of the system under test (SUT) using the camera in the first section and use an image recognition function to match a pattern between the first image and a second image referenced in the first data from the fourth section. A second plurality of machine readable computer processing instructions in the fifth section may be operable to display a plurality of three-dimensional models from the first data in the fourth section by using an augmented reality function to display the second image of the system under test (SUT) and a third image of the plurality of subsystems under test wherein the system under test (SUT) is represented in the third section. A third plurality of machine readable computer processing instructions in the fifth section are operable to identify a system under test (SUT) using the first section to stream a first video of the system under test (SUT) using the camera in the first section. A fourth plurality of machine readable computer processing instructions in the fifth section may be operable to encode a first network message or a first video stream in the first section using a pseudo-random encryption key. A fifth plurality of machine readable computer processing instructions in the fifth section may be operable to communicate to the sixth section through a wired or wireless network communication method. A sixth plurality of machine readable computer processing instructions in the fifth section may be operable to display a first visual overlay with a plurality of designators to be displayed on the third section as information to be received from the sixth section. A seventh plurality of machine readable computer processing instructions of the fifth section may be operable to display a second visual overlay with a plurality of maintenance procedures, a plurality of maintenance videos, a plurality of sensor parameters on subsystems under test for the system under test (SUT) by issuing a plurality of machine readable computer processing instructions to the sixth section.
The sixth section is illustratively in a first network including a plurality of local nodes comprising the first section, a local switch node connected to the sixth section and connected to at least one of a plurality of switch nodes for a plurality of subsystems under test, a plurality of switch nodes for a plurality of external sensors, and a plurality of switch nodes for a plurality of remote locations. A data transfer mechanism from a local switch node in the first network to a plurality of switch nodes for a plurality of subsystems under test is deactivated. A data transfer mechanism from a local switch node in the first network to a plurality of switch nodes for a plurality of external sensors is deactivated.
The plurality of machine readable computer processing system instructions in the sixth section may include an eighth plurality of machine readable computer processing instructions operable to process a first video stream received from the first section into a plurality of video formats used for compression and transfer. A ninth plurality of machine readable computer processing instructions in the sixth section may be operable to collect and store a plurality of sensor parameters from a plurality subsystems under test in a system under test (SUT) or plurality of external sensors into a second storage section in the sixth section. A tenth plurality of machine readable computer processing instructions in the sixth section may be operable to authenticate user access from a plurality of remote locations. An eleventh plurality of machine readable computer processing instructions in the sixth section may be operable to provide prognostics and health management services to evaluate a plurality of sensor parameters stored in the second storage section for potential failure and suggested action by comprising a statistical recognizer or a machine learning recognizer including one of: a regression recognizer, a Hidden Markov Model (HMM) recognizer, a dynamic time warp (DTW) recognizer, a neural network, a fuzzy logic engine, a Bayesian network, an inference rule engine, and a trajectory similarity based prediction (TSBP). A twelfth plurality of machine readable computer processing instructions in the sixth section may be operable to compress data stored in the second storage section into a smaller data size format to provide bidirectional transfer to a plurality of remote locations. A thirteenth plurality of machine readable computer processing instructions in the sixth section may be operable to encode a second network message or a second video stream stored in the second storage section located in the sixth section using a pseudo-random encryption key. A fourteenth plurality of machine readable computer processing instructions in the sixth section may be operable to update data in the fourth section and the sixth section from a data library at a remote location.
According to another illustrative embodiment of the present disclosure, a non-transitory computer processing medium includes a plurality of machine readable processing instructions including a first plurality of machine readable computer processing instructions operable to identify a system under test (SUT) using a first section to capture a first image of the system under test (SUT) using a first camera in the first section and use of an image recognition function to match a pattern between the first image and a second image referenced in a first data from a fourth section, and a second plurality of machine readable computer processing instructions operable to display a plurality of three-dimensional models from the first data in the fourth section by using an augment reality function to display the second image of a system under test (SUT) and a third image of a plurality of subsystems under test wherein a system under test (SUT) is comprised of in the third section. A third plurality of machine readable computer processing instructions are operable to identify a system under test (SUT) using the first section to stream a first video of the system under test (SUT) using the first camera in the first section. A fourth plurality of machine readable computer processing instructions are operable to encode a first network message or a first video stream in the first section using a pseudo-random encryption key. A fifth plurality of machine readable computer processing instructions are operable to communicate to the sixth section through a wired or wireless network communication method. A sixth plurality of machine readable computer processing instructions operable to display a first visual overlay with a plurality of designators to be displayed on the third section as information to be received from the sixth section. A seventh plurality of machine readable computer processing instructions operable to display a second visual overlay with a plurality of maintenance procedures, a plurality of maintenance videos, a plurality of sensor parameters on subsystems under test for the system under test (SUT) by issuing a plurality of machine readable computer processing instructions to the sixth section. An eighth plurality of machine readable computer processing instructions are operable to process a first video stream received from the first section into a plurality of video formats used for compression and transfer. A ninth plurality of machine readable computer processing instructions operable to collect and store a plurality of sensor parameters from a plurality subsystems under test in a system under test (SUT) or plurality of external sensors into a second storage section in the sixth section. A tenth plurality of machine readable computer processing instructions operable to authenticate user access from a plurality of remote locations. An eleventh plurality of machine readable computer processing instructions operable to provide prognostics and health management services to evaluate a plurality of sensor parameters stored in the second storage section for potential failure and suggested action by comprising a statistical recognizer or a machine learning recognizer including one of: a regression recognizer, a Hidden Markov Model (HMM) recognizer, a dynamic time warp (DTW) recognizer, a neural network, a fuzzy logic engine, a Bayesian network, an inference rule engine, and a trajectory similarity based prediction (TSBP). A twelfth plurality of machine readable computer processing instructions operable to compress data stored in the second storage section into a smaller data size format to provide bidirectional transfer to a plurality of remote locations. A thirteenth plurality of machine readable computer processing instructions operable to encode a second network message or a second video stream stored in the second storage section located in the sixth section using a pseudo-random encryption key. A fourteenth plurality of machine readable computer processing instructions to update data in the fourth section and sixth section from a data library at a remote location.
According to a further illustrative embodiment of the present disclosure, an interactive remote maintenance assist system includes a system under test (SUT) including one or more system racks having a plurality of computing servers disposed therein, the computing servers being comprised of a plurality of circuit boards wherein at least one circuit board includes one or more sensors configured to provide data indicating one of temperature, fan speed, processor speed, memory capacity, and voltage power values, the data corresponding to performance parameters of at least one computing server. A first display device is positioned at a first location, the first display device including a first graphical user interface (GUI) configured to capture and display a visual graphic, the visual graphic depicting a model of the system under test. A second display device is positioned at a second location that is spaced apart from the first location, the second display device being communicably coupled to the first display device and including a second (GUI) configured to display the visual graphic depicting the model of the system under test (SUT) being displayed by the first display device. A maintenance server is communicably coupled to the first display device and the second display device, the maintenance server being disposed intermediate the first and second display devices and being configured to provide bi-directional data communication between the first and second display devices. The first and second displays each include object recognition logic such that the model of the system under test (SUT) depicted by the first and second GUIs each include digital relational structures corresponding to the computing servers and wherein the data provided by the sensors is viewable within the model and accessible by at least the second display device. The second display device further includes logic configured to perform one of regression and statistical analysis on the data provided by the sensors to enable a user at the second location to determine one of a current and prospective system fault associated with at least one computing server and wherein the maintenance server enables the user to transmit one or more instructions for resolving the current system fault and for mitigating the occurrence of the prospective system fault.
According to another illustrative embodiment of the present disclosure, a method of operating an interactive remote maintenance system includes providing a system under test (SUT) including one or more system racks having a plurality of computing servers disposed therein, the computing servers having a plurality of circuit boards including at least one sensor that provides data indicating one of temperature, fan speed, processor speed, memory capacity, and voltage power values, the data corresponding to performance parameters of at least one computing server, providing a first display device positioned at a first location, the first display device including a first graphical user interface (GUI), providing a second display device positioned at a second location that is spaced apart from the first location, the second display device being communicably coupled to the first display device and including a second (GUI), and providing a maintenance server communicably coupled to the first display device and the second display device, the maintenance server providing bi-directional data communication between the first and second display devices. The method further includes collecting, by the first display device, the data corresponding to performance parameters of the at least one computing server and transmitting the data to the second display device by way of the maintenance server. The method further includes identifying, by the first display device, one or more sensor data types corresponding to the cause of a current system failure and a prospective system failure, wherein identifying the one or more sensor data types includes utilizing principle component analysis logic. The method also includes creating, by one of the first display device and the maintenance server, a regression trend based on a linear regression method, wherein the regression trend includes a performance trend line indicating prospective performance of the system under test. The method further includes comparing, by one of the first display device and the maintenance server, the collected performance parameters to the performance trend line of the regression trend. The method also includes predicting, by one of the first display device and the maintenance server, prospective values of the at least one sensor, the prospective values including data indicating one of temperature, fan speed, processor speed, memory capacity, and voltage power values, the data corresponding to prospective performance parameters of the at least one computing server. The method further includes displaying, by the first and second display devices, a visual graphic depicting a model of the system under test, the model including digital relational structures corresponding to the computing servers and wherein the data provided by the sensors is viewable within the model and accessible by at least the second display device. The method also includes determining, by one of the first display device and the maintenance server, a root-cause for one of the current system failure and the prospective system failure associated with the at least one computing server and wherein the maintenance server enables a user at the second location to transmit, to the first display device, instructions for resolving the current system failure and for mitigating the occurrence of the prospective system failure, the instructions being displayed by way of the first GUI.
Additional features and advantages of the present invention will become apparent to those skilled in the art upon consideration of the following detailed description of the illustrative embodiment exemplifying the best mode of carrying out the invention as presently perceived.